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SUMMARY 


Consistency  of  the  autoregressive  method  of  density  estimation. 

abbreviated  title:  autoregressive  estimator  of  density. 

A density  function  f(’)  , with  f and  log  f(*)  both 

Lebesgue-integrable,  has  a representation  as  an  autoregressive  spectral 

density.  We  use  this  representation  to  obtain  new  density  autoregressive 

estimators  of  different  orders  p based  on  the  empirical  characteristic 

function  of  a sample  of  size  n . We  prove  the  consistency  of  these  new 

3 -1 

estimators  as  11m  p n =0  under  varying  conditions  on  the  smoothness 


, n-*«> 

of  f(-)  . 


1 

CONSISTENCY  OF  THE  AUTOREGRESSIVE  METHOD 
OF  DENSITY  ESTIMATION 

by 

Jean-Pierre  Carmichael^ 

State  University  of  New  York  at  Buffalo 

1.  Introduction 

The  autoregressive  method  has  been  used  so  far  only  in  the  context 
of  time  series.  Consider,  for  instance,  a discrete  time  real  process 
{X(t),  t e Z}  (where  Z is  the  set  of  all  integers)  with  stationary 
covariance  function 

Cov  ^X(t) , X(t  + v)^  = R(v)  , V e Z . 

X(* ) is  an  autoregressive  process  of  order  p if  there  exist  a sequence 

{Ojp  , j = l,...,p}  and  an  orthogonal  process  {ri(t)  , t e Z}  with  mean 

2 

zero  and  variance  a >0  such  that 

n 

P 

(1.1)  X(t)  + E a.  X(t-j)  = n(t)  , teZ  . 

j=l 

When  (1.1)  holds,  R(*)  satisfies  the  Yule-Walker  equations 

P 

(1.2)  E cij„R(Jl-j)  = 0 , j = 1.....P 

1 
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(1.3) 


r a.  R(S.)  = 

£-0  ^ 


with  a 


Op 


In  the  autoregressive  method,  we  obtain  estimates  {a.  , j = 

a.2 

and  from  a sample  {X(l) , , . . ,X(T) } by  solving  (1.2)  and  (1.3)  with 


R(*)  replaced  by  R^(*) 


„-l 


T-  V 


R_.(v)  = T ^ I X(t)  X(t  + v) 
t=l 


The  spectral  density  f(*)  of  the  process  X(.)  is  defined 
Implicitly  by 


R(v)  = / e^'^’'f(x)  dx 

-TI 


For  an  autoregressive  process  of  order  p , 


(1.4) 


f (x)  = 0^(271)  ^ II  + !■  a xeC-n.Tt] 

P n j.l  JP 


It  is  estimated  using  the  estimated  autoregressive  parameters. 

The  consistency  of  these  procedures  has  been  studied  by  Kromer 
(1969)  and  Berk  (1974). 

In  this  paper,  we  present  the  first  application  of  the  autoregres- 
sive method  in  the  context  of  indapendent  identically  distributed  random 
variables  as  a new  method  of  density  estimation  and  we  study  its  conver- 
gence properties  in  terms  of  consistency. 


2. 
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L 


Probabilistic  interpretation  of  the  autoregressive  method. 


Let  X be  a bounded  random  variable  taking  values  in  [-it, it] 
with  absolutely  continuous  distribution  function  F(*)  , density  function 
f(»)  and  characteristic  function  ({)(•)  , 


♦(v)  = / f(x)  dx  , V real 

-TT 

It  follows  that  $(•)  is  llermitlan  and  nonnegative  definite,  i.e.  (})(•) 
satisfies  the  necessary  and  sufficient  conditions  to  be  a covariance 
function.  It  also  follows  that  f(*)  is  completely  determined  by  the 
sequence  {(}>(v)  , v = 0,1,2,...}  (see  Feller  (1966),  chapter  19). 

We  think  of  4>(*)  the  covariance  function  of  a stationary  com- 
plex time  series  (unobservable)  and  of  f(*)  as  its  associated  spectral 

density. 

# 

Theorem  2.1 

If  f ^(•)  and  log  f(*)  are  both  Lebesque-integrable  on  [-tt.tt]  , 
then  f(*)  can  be  represented  as  the  spectral  density  f(*)  of  an 
infinite  order  autoregressive  process: 


(2.1) 

4>(v)  = 

TT  . ^ 

1 e 1 (x)  dx  , V c Z 

-n 

(2.2) 

X 

tt 

.00  . o 

a (2ti)  1 + E a.  e 

“ j-1  J 

(2.3) 

00 

E |a. 
j-'l  ^ 

1^  < CO  , 0^  > 0 

* ' 00 

(2. A) 

f (x)  “ 

f (x)  , a.e. 

I 

I 

I 
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Proof. 

Apply  the  argument  of  Doob  (1953,  p.  577)  to  f(*)  and  f”^(«)  . 

O 

We  approximate  f(*)  by  fp(^)  , the  spectral  density  of  an  auto- 
regressive process  of  order  p such  that 


TT  , ^ 

(2.5)  (j)(v)  = / e '^f  (x)  dx  , V = 0,  ± 1 ±p 

-n  ^ 

p 2 

(2.6)  f (X)  = al  (2v)-^\l  + E a.  . 

P P JP  ' 

Now,  we  can  estimate  {a  , j = l,...,p}  and  for  different  values 

JP  P 

of  p as  in  Section  1 by  solving  the  Yule-Walker  equations  with  (^(•) 

being  estimated  by  the  sample  characteristic  function  <ti  (*)  of  a 

n 

sample  {X, , . . . ,X  } 

i tv 


n ivXj^ 

<J>  (v)  = n Z e 
k=l 


Finally,  the  estimated  density  is  » 


f (x)  » ol  (2Tr)‘^  |l  + E a 
P P ' jp  ' 


-2 


Note  that  4>jj(*)  is  usually  a complex-valued  function  and  the  estimated 
autoregressive  coefficients  are  also  complex. 

Theorem  2.2 

fp(*)  is  a probability  density  function. 
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Proof. 


By  definition  of  » 


^ f (x)  dx  , v = 0,  ± 1, . . . , ± p . 

-IT  ^ 

'' 

For  V = 0 , / f (x)  dx  = 1 . Finally,  o > 0 (Pagano  (1973)). 

-IT  P I 


3.  Approximation  theory  interpretation  of  the  autoregressive  method. 

Let  F(*)  be  a nondecreasing  bounded  function  with  infinitely  many 

points  of  increase,  defined  on  . We  denote  by  the  space 

of  measurable  complex-valued  functions  u(*)  such  that 

/ |u(e  ’')!  dF(x)  < « . It  is  well  knowi  that  with  the  inner 

-IT  F 

product 

(3.1)  ^u(*).v(O^P  = / uCe^’')  v(e^'^)  dF(x) 

2 

u(*)  and  v(*)  in  L 

F 

is  a Hilbert  space. 

2 2 

From  the  set  of  powers  {l,z,z  ,...}  in  L^.  , we  obtain  a set  of 
orthonormal  polynomials  {gQ(* ) ) ,82(* ) , • . • } uniquely  determined  by 


(3.2) 


and 


£.-1 

gj(z)  «=  E aj^  z , a^p  > 0 , for  all  I 


j=0 


0£ 
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(3.3)  , for  all  j and  k . 

Consider  the  subspace  of  generated  by  ^gQ(*  ) ) . — .gp(*)^ 

It  is  a reproducing  kernel  Hilbert  space,  i.e.  there  exists  a function 
Kp(*,*)  of  two  complex  variables  such  that 


(3.  A) 


Kp(*,y)  c Lp  , for  any  fixed  y 


(3.5) 


(3.6) 


K (‘.y)  = E 
p j=o  JP  J 


^u(*).  Kp(*  ,y)^p  = u(y)  , for  all  u(*)  e 


In  fact,  K (• ,y)  = E g.(y)  g.(*)  • We  restrict  our  attention  to 

P j=0  ^ J 

Kp(*,0)  and  express  it  as  a polynomial 


K (z,0)  = I b z-^ 


Let  Uj(z)  = , j = 0,1, ,p  . By  the  reproducing  property  of 

Kp(*,*)  , we  have  that 

(3.7)  j = 1.---.P 


(3.8)  ^ 


If  We  Introduce  the  notation  <()(•) 


♦(v)  “ I e ^ dF(x) 
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then  the  system  of  equations  (3.7)  and  (3.8)  becomes 


P 


(3.9) 

2 b d.(j-£)  = 0 , 
£=0 

P _ 

(3.10) 

^ b <♦.(-£)=  1 . 

£=0  P 

Upon  taking  complex  conjugates  and  identifying  4l(•)  with  R(*)  , we 
see  that  (3.9)  and  (3.10)  are  equivalent  to  (1.2)  and  (1.3)  up  to  a con- 
stant factor,  the  difference  being  that  = 1 . Thus  we  divide  (3.9) 

and  (3.10)  by  Kp(0,0)  and  make  the  following  identification  between  the 
two  systems  of  equations; 


from  which  identification  follows  that 

fpM  = 0^(2tt)"^|i  + = Kp(0,0)(2TT)"^|Kp(e‘’',0)|  ^ . 

We  have  thus  proved  the  following  theorem. 

Theorem  3.1 

If  F(*)  is  an  absolutely  continuous  distribution  function,  f(*) 
Its  density,  f (•)  and  log  f(*)  are  Lebesgue-integrable  and  we 
replace  In  (3.9)  and  (3.10)  (J>(*)  by  the  sample  characteristic  function 
(()n(*)  as  defined  in  Section  2,  we  obtain  two  representations  for  f (•) 


I 


Cl 
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(3.11) 


fp(x)  = 


''2  -1 

0^(2^) 


1 + 


P 

Z 

j=l 


a.  e 
jp 


ijx 


Kp(n,0)(27T) 


-1 


Kp(e^*,0) 


The  properties  of  K^C*,©)  have  been  studied  extensively  in  the 
approximation  theory  literature,  e.g.  Grenander  and  Szego  (1958), 

Geronimus  (1961).  We  quote  known  results  from  these  sources  without 
proof  as  we  need  them. 

We  study  the  convergence  properties  of  the  autoregressive  method 
under  the  following  sets  of  conditions: 

Conditions  A : 

F(*)  is  an  absolutely  continuous  distribution  function  with 
infinitely  many  points  of  increase,  defined  on  [-7r,7r]  . 

£(•)  is  the  corresponding  density  function. 

f ^(•)  and  log  f(*)  are  Lebesgue-integrable  on  [-tt.tt]  . 

Conditions  B : 

Conditions  A are  satisfied.  Furthermore,  0 < m < f (x)  < M < “ , a.e. 


f(*)  e Lip  (^,2) 


where 


Lip  (a, 2)  = |u(*)  : sup  ( / (u(x  + h)  - u(x)l^  dx)  = 0 (f'^)'|  . 

k |h|<6'-Ti  J 


Conditions  C : 

Conditions  A are  satisfied.  Furthermore,  0 < m < f (x)  < M < oo  , a.e. 


£(•)  = d(*)  , a.e. 


d(*)  e Lip  (a, 2)  , a > *5 
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4.  bias  stud\ 


In  the  estimation  of  £(•)  , we  have  used  an  approximate  represen- 

tation  f (•)  that  we  have  estimated  by  f (•)  . In  this  section  we 
P P 

study  how  good  approximation  to  f(*)  • 


Lemma  4.1 


Define  gp(*)  by 


(4.1) 


g*(z)  = Kp(z,0)  Kp(0,0) 


The  Lebesgue-integrability  of  log  £(•)  is  a necessary  and  sufficient 
condition  for  the  existence  of  the  following  limits 


(4.2) 


0 < lim  K (0,0)  = lim  E lg,(0)l  = E |g,(0)r  = K(0,0)  < - ; 


(we  will  now  use  = Kp(0,0)  and  K = K(0,C)  ) 


(4.3) 


lim  g*(z)  = g(z)  = K E g.(0)  g.(z)  , |zl  < 1 

p-x»  P j=l  J -* 


where  the  convergence  is  uniform  in  |z|  < r < 1 ; 


(4.4) 


g(e^’')  = lim  gCre^’')  , a.e.; 

r-»l" 


(4.5)  f(x)  = (2Tr)"^|g(e^’')  I , a.e.; 


for  E •=  {x  : 0 < f(x)  < <»}  , define  g(*)  , 
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g(x) 


^ 1.x 

g(e  ) , X € E 


X ^ E 


g has  the  following  expansion  in  terms  of  the  orthonormal  polynomials 
{gj(*)  . j = 0,1.2,...} 


(4.6) 


g(x) 


~ K ^ 


Z g (0)  g.Ce^’') 
j=0  ^ J 


that  converges  in  Lp  ; 


(4.7) 


11m  lli(-)  - kJ  g*(-)||p  = 0 

p-KO  ^ ^ 


/ TI  Iv  2 N'-J 

where  ||u(*)||p  = f / |u(e  )|  f(x)  dxj  (Geroniraus  (1961),  Chapter  II). 


Theorem  4 . 2 

Under  Conditions  A , 


(4.8) 


(4.9) 


Proof : 


" —*1  1 

11m  j |f  (x)  - f (x)  I f(x)  dx  = 0 
p-w  -TT  ^ 


11m  / |f(x)  - f (x)  I f ^(x)  dx  = 0 

P-»oo  -TT  P P 


|r\x)  - f ^X)|  = 2TT||g(e^’')|^  - |g*(e^*)|^|  , a.. 
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Thus, 


/ ir\x)  - f-\x)i  f(x)  dx 

•IT  ^ 


^ It,  Up(e^'')|)(U(e^’')  - gpCe^*)!^  f(x)  dx 


2^(l|8(*)l|p  + l|8*(-)||pj!|g(.)  - g 


p'  '"F 


by  Schwarz  Inequality. 
llKp(*)|lF  = 1 » Il8(-)|lp  = 1 (by  (4.6))  and 

11m  ||g(.)  - g*(.)  II  = lin,  ||g(.)  - g*(-)  lip  = 0 by  (4.7  and  (4.2) 

p-Ho  ^ r»-w«  r ^ 


p-^oo 


Finally,  (4.9)  is  equivalent  to  (4.8). 

Lemma  4 . 3 

Under  Conditions  B , 

ISj  (z)  I < C , for  |z  I si 
Wc  can  replace  the  Lipschitz  condition  by 


<>(v)  ■=  0(v  ^) 


(Geronimus  (1961),  Theorem  3.8). 


F 


I 

I 


1 
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Theorem  4.4 

Under  Conditions  B , 


(4.10) 

lim 

77 

/ 

|f'^(x) 

P+OO 

-77 

77 

(4.11) 

lim 

/ 

|f(x)  - 

p>>oo 

-77 

,2  ^2 
P P 


Proof. 


As  in  the  proof  of  Theorem  4.2, 


|f  ^(x)  - fp^(x)|  < 277  |g(e^’')|  + |g*(e^’')l  jgCe^’')  - g*(e^^)  | , but 


IgCc*’^)]  < m ^ a.e.,  as  f(*)  > m > 0 , a.i 


|gp(e^^)|  < C , by  Lenuna  4.3. 


Thus, 


J |f  ^(x)  - fp*^(x)|  dx  < 47i^(m  ^ + C)^  ra  ^ IjgC*)  - 8p(*)Il  p 

-71 

and  the  right-hand  side  converges  to  zero  as  in  Theorem  4.2.  Again, 
(4.10)  and  (4.11)  are  equivalent,  f(*)  being  bounded  from  below  a.e. 

□ 

Lemma  4.5 

Under  Conditions  C , 

ff 

lim  g (e  ) = g(e  ) , uniformly  on  [-77,  77]  . 


(Geronimus  (1961),  Theorem  5.2). 
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Theorem  A. 6 

Under  Conditions  C , 


(4.12) 


'w  2 

lim  / |f(x)  - f (x) I dx  = 0 


-IT 


(4.13) 


lim  f (x)  - (27f)  ^ lg(e^^)  | uniformly, 

p+oo  P 


Proof. 


Under  Conditions  C , 0 < a < |gp(e^’')  | < A < “ for  any  p 


(see  Ibragimov  (1964),  Lemma  5),  and  because  |gp(e^’')  I = IgpCe^’')  I , 


it  follows  that  0 < b < fp(x)  < B < “ and  (4.12)  follows  from  (4.10). 


(it  ix  I ^ I ix 
8_(c  )|  = |g(e  )|  uniformly  by  Lemma  4.5  and  the  fact 


* ^ 

that  !?_(*) I Is  uniformly  bounded  from  above  and  below. 

P p 


Having  obtained  uniform  pointwise  convergence,  we  state  the  follow- 
ing rates  of  decrease  of  the  bias. 


Theorem  4 , 7 

Under  Conditions  C , 


|f(x)  - fp(x)l  = 0(p”^)  a.e.,  0s6<a-!s 


Moreover  if  the  function  d(*)  has  r derivatives  and  d^’^^(*)  c Lip  (a, 2)  , 
0 < a s 1 , then 


O 

|f(x)  - fp(x)|  = 0(p“  ) , a.e.,  0^3<r  + “-  *s 


(Kromer  (1969),  Theorem  3.12). 
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Proof . 


These  results  are  well-known.  For  a proof  of  (5.4)  see  Davies  (1973) 


Lemma  5 . 2 

Let  0 < m < f(x)  5 M < “ , a.e.  . 

IT 

Let  (t(v)  = / e^'^  f(x)  dx  , v=0.  ±1,±2, 

-n 

Define  the  Hennitian  Toeplitz  matrix  R . 


(t)(0)  . . . <{>(p  - 1) 

• • 

• • 

• • 

<t)(l-p)  — <})(0) 


Then,  Rp  is  nonsingular. 


(5.5) 

and 

(5.6) 

(5.7) 


2tt  m < X . (R  ) < X^^^(R^)  < 2tt  M 
min  p max  p 


lim  X , (R  ) = 2Tr  m 
min  p 

p-><*> 


lim  X (R  ) = 2it  M 
max  p 

p-H»  ^ 


(Grenander  and  Szego  (1958),  Chapter  5). 


Lemma  5.3 


r 

~P 


f<()(-l),...,4'(-P))'  • If  limp^n"^-0,  then, 
\ ' n-x» 


|j^  - tpll  converges  to  zero  in  probability. 
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Proof. 


lip  ■ IplI  - max(|({)^(v)  - 4)(v)|  ; v = -1 -p) 


P^/p”  inax(  |(t)^(v)  - (})(v)  I ; V = -1, . . . ,-p)  S ^ 1 - n”^  1^) 


by  Bonferroni's  inequality  and  Chebyshev's  inequality. 


Corollary  5.3.1 

2 “1  ^ 

If  lim  p n = 0 , II “ ^pll  converges  to  zero  in  probability. 

3 “X  ^ ^ \r  ^ 

If  lim  p n = 0 , P^HXp  ~ £pll  P^H^p  ~ ^pH  t>oth  converge 

to  zero  in  probability. 


Proof. 


Just  note  that  because  R and  R arc  Tocplitz  and  Hermitian, 

P P 


l*^p  “ •^pll  - ^ ^ max^l(5^(v)  - ({)(v)|  , V = -l,...,-p^ 


Lemma  5. A 


cip  - (aQp.....cipp,0....) 

Ct  “ * ^2^  >••••)  • 

If  0 < m 5 f (x)  < M < 00,  a*e.  [-tt.tt]  , then 


lISp  “ s!l  = 0 

p-x»  ^ 
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Proof. 


By  the  representation  introduced  in  Section  3 and  Leirana  4.1, 


and 


Z a. 


j=0 


K g(e  ) , a.e. 


It  follov;s  that 


ISp  - all^  = - “i)  dx 

P -Trj=l  J 


5 g*(-) 


p'  " "F 


0 , by  Lemma  4.1. 


p->oo 


Theorem  5.5 


If  _0  < m S f(x)  < M < “ , a.e.  and  lira  n ^ = 0 , then 

1^+00 

- 3p||  and  [K^  - Kp | converge  to  zero  in  probability. 


Proof. 


It  Is  sufficient  to  consider  a •=  (a a ) 

~p  Ip’  pp-' 


yv  ^ I /V 


'[( 


flp  - Sp  ' K \(l.-  L^)  (K  - K) 


p ~p  ~p'  p p' 


(5.8) 
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because  the  Yule-Walker  equations  can  be  written 


R a 
P ~P 


R a 
P ~P 


(5.9) 


ISp-Spll^llRp 


r + R -R 
~P  P P 


l%ll] 


by  (5.2) 


We  now  bound  each  terra  on  the  right-hand  side. 

A 

lira  l)a  11  = ))oi))  < “ as  can  be  seen  from  Lemma  5.4.  Both  Hr  - X !l 

p->oo  P P P 

Ak 

and  l|Rp  - Rp||  converge  to  zero  in  probability  by  Lemma  5.3  and  its 
Corollary. 


Finally,  we  use  (5.4)  to  bound  [[r  . Note  that  R is  non- 

A 

singular  and  because  |jR^  - R^d  converges  to  zero  in  probability, 


|Rp  - Rpll  Ml  - e)  llRp^li;;^  ^ (1  - e)  2ti  M 


so,  by  (5.4),  llRp^lln  s e llYp^ll^  < e(2TTra)"^  , 


l.e.  ||Rp  lly  is  bounded  with  probability  one. 

To  prove  the  second  part  of  the  theorem,  we  note  that 
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P - 


'P  ■ ‘‘P  ' L “jp*n«>  - 


j = l 


j = l 

<{>(j))  + 4>(j)(a 


. -C.  ,] 

JP  jpj 


1%^  ■ ^ ll“pll  p •naxCkjjCj)  - («>(j)|  , j = l,...,p) 


+ Il2p  - «pll 


j=l 


This  goes  to  zero  in  probability  provided  lim  Z |<{)(j)|  < “ . 

p+oo  j = l 

P 2^9  9 

lim  Z |<*>(j)|  = / f"^(x)  dx  < 2tt  < » 

p+«>  j=l  -Tf 


because  {({)(j)}  are  the  Fourier  coefficients  of  f(*)  . Finally, 

A 

|K  - K H converges  to  zero  in  probability  as  K < K < “>  . 

P P P □ 


Corollary  5.5.1 
Let 


But 


~P 


‘“ip “pp'“'---> 


2p  ■ ‘“ip “pp-“---> 

Cl  “ ^^1  * ^2  • * * * ^ 


Under  the  conditions  of  Theorem  5.5,  HOp  - a|I  and  - k|  converge 

to  zero  in  probability. 


< 
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Proof. 

I ~ - I “ ^pl  I ^p  “ *^1  ' Similarly 

Il2p  -sll^  lISp  -“pll+IISp  -sl|.  . 

Just  apply  Lemma  4.1,  Lemma  5.4  and  Theorem  5.5. 

□ 

6.  Consistency  of  f (• ) 

Lemma  6 . 1 

If  0 < m < f(x)  < M < « , a.e.  and  lim  p^  n"^  = 0 , then 
ix  * • • 

|8p(e  ) - gp(e  ^)|  converges  to  zero  in  probability  uniformly  in  x 


Proof. 


By  the  same  technique  as  in  Theorem  5.5 


- s>-„  . A Kp-^plMK^K^I  ? K 


j=l 


p p'  j=o  ■ jP' 


•^P  ^ Il2p  - 2pll  + |K^  - • ||a,l|  . 


p p ' 


lor  aiJL 


*5  i, 

P converges  in  probability  to  ; 


in  probability  if  lim  p^  n“^ 

/T|£^-k^I 


•A  llssp  - Op 


converges  to  zero 
by  Theorem  5.5,  and  the  same  for 
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We  can  now  prove  consistency  theorems  analpgous  to  Theorems  4.2, 
4.4  and  4.6.  We  give  only  one  example. 


Theorem  6.2 


Under  Conditions  C and  if  lim  p^  n~^  = 0 , then 


f“^(x)  - 2tt  |g(e^’')|^|  and  |f  (x)  - (2ti)“^  |g(e^’')|"^ 


converge  to  zero  in  probability  uniformly  in  x 


Proof. 


We  prove  only  the  first  statement. 


|fp\x)  - 2tt  |g(e*’')|^|  < |fp^(x)  - f ^x)|  + |f~^(x)  - 27T  |g(c^'')|^| 


M - 271  |g(e^^)|^[  = 0 , uniformly  in  x , by  Theorem  4.6.  On 

p-40O  ^ 

the  other  hand, 


^ 27T(|g*(e^'')|  + Igp(e^’')l)lg*(e^’') 


*,  ix. 
Bp(e  ) 


We  now  apply  Lemma  4.5  and  Lemma  6.1. 
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